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CN Abstract 

q This paper deals with a universal coding problem for a certain kind 

of multiterminal source coding system that we call the complementary 
delivery coding system. In this system, messages from two correlated 
C*~) sources are jointly encoded, and each decoder has access to one of the two 

messages to enable it to reproduce the other message. Both fixed-to-fixed 
length and fixed-to- variable length lossless coding schemes are considered, 
r. Explicit constructions of universal codes and bounds of the error proba- 

I | bilities are clarified via type-theoretical and graph-theoretical analyses. 

^ Keywords: multiterminal source coding, complementary delivery, univer- 

sal coding, types of sequences, bipartite graphs 

1 Introduction 

qq The coding problem for correlated information sources was first described and 

t-H investigated by Slepian and Wolf [I], and later, various coding problems derived 

from that work were considered (e.g. Wyner [3J, Korner and Marton [3], Sgarro 
[4]). Meanwhile, the problem of universal coding for these systems was first 
y—i investigated by Csiszar and Korner [5 . Universal coding problems are not only 

interesting in their own right but also very important in terms of practical 
applications. Subsequent work has mainly focused on the Slepian- Wolf coding 
J> system [SI [5] since it appears to be difficult to construct universal codes for 

most of the other coding systems. For example, Muramatsu [5] showed that we 
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cannot construct asymptotically optimal fixed-to- variable length (FV) universal 
code for Wyner-Ziv coding systems [TU] in terms of the coding rate. 

This paper deals with a universal coding problem for a certain kind of mul- 
titerminal source coding system that we call a complementary delivery coding 
system [IT] . Figure [l] shows a block diagram of the complementary delivery 
coding system. The encoder observes messages emitted from two correlated 
sources, and delivers these messages to other locations (i.e. decoders). Each 
decoder has access to one of two messages, and therefore wants to reproduce 
the other message. Although the previous articles [11] considered lossy con- 
figurations, this paper considers a lossless configuration. We show an explicit 
construction of fixed-to-fixed length universal codes. We also clarify the up- 
per and lower bounds of the error probabilities via type-theoretical and graph- 
theoretical analyses. Fixed-to-variable universal codes can also be constructed 
in a similar manner. 

This paper is organized as follows: Notations and definitions are provided in 
Section [2j Previous results for the complementary delivery coding system are 
shown in Section [3] A proposed coding scheme is described in Section [4] Several 
coding theorems are clarified in Section [5] Variable- length coding is discussed 
in Section [6] Finally, concluding remarks are given in Section [7] 

2 Preliminaries 

2.1 Basic definitions 

Let A" be a finite set, B be a binary set, and B* be a set of all finite sequences 
in the alphabet B. Let \X\ be the cardinality of X and Tm — {1,2, •• • , M}. 
A member of X n is written as x n — X2, • • • , x n ), and substrings of x n are 
written as x\ = (x t , Xj+i, • • • , Xj) for i < j. When the dimension is clear from 
the context, vectors will be denoted by boldface letters, i.e., x S X n . M(X) 
denotes the set of all probability distributions on X. Also, A4(X\Py) denotes 
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the set of all probability distributions on X given a distribution Py £ M(y), 
namely each member 

P x \y of M(X\P Y ) is characterized by P XY £ M(X x y) 
as Pyy = Px\yPy- A discrete memoryless source (X,Px) is an infinite se- 
quence of independent copies of a random variable X taking values in X with a 
generic distribution Px £ Ai(X). We will denote a source (X, Px) by referring 
to its generic distribution Px or random variable X . For a correlated source 
(Jf, Y), H(X\Y) denotes the conditional entropy of X given Y. For a generic 
distribution Py S M(y) and a conditional distribution Px\y € ■M(A'lPy), 
•ff(-Px|y|-FV) also denotes the conditional entropy of X given Y. D(P||Q) de- 
notes the Kullback-Leibler divergence between two distributions P and Q. In 
the following, all bases of exponentials and logarithms are set at 2. 

2.2 Types of sequences 

Let us define the type of a sequence x G as the empirical distribution Q x £ 
■M(A") of the sequence x, i.e. 

Q x (a) = -N(a\x) Va € A", 
n 

where iV(a|a;) represents the number of occurences of the letter a in the sequence 
x. Similarly the joint type Q x ,y E M(X x y) is defined by 

Q*, v {a,b) = -N(a,b\x,y) V(a,b)eXxy. 
n 

Let V n (X) be the set of types of sequences in X" . Similarly for every type Q S 
V n (X), let V„(y|Q) be the set of all stochastic matrices V : X — > y such that for 
some pairs (x,y) € X n x y n of sequences we have Q x . y (x,y) = Q(x)V(y\x). 
For every type Q £ P n (X) we denote 

T£ d =- {xeX n :Q X = Q}. 
Similarly for every x £ Tq and V £ V n (y\Q) we define 

T^(x) A ^{y £ y n : 

Q(x)V(y\x) = Q x , y (x,y), V(x,y) £ X x y}. 

Hereafter, we call Ty(x) a V-shell. 

Here, let us introduce several important properties of types. 

Lemma 1. (Type counting lemma [12j Lemma 2.2]) 

The number of different types of sequences in X n is less than (n+l)' x ' , namely 

\V n (X)\ < (n + l)l*L 
Lemma 2. (Sizes of V-shells (TSJ Lemma 2.5]) 

For every sequence x £ X n and every stochastic matrix V : X — > y such that 
the corresponding V-shell T v (x) is not empty, we have 

\T v (x)\ > (n + l)-\ x \\ y \eMnH(V\P x )}, 
\T v (x)\ < exp{nH(V\P x )}. 
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Lemma 3. (Probabilities of types [T3J Lemma 2.6]) 

For every type Q of sequences in X n and every distribution Px on X , we have 
P x (x) = exp{-n(D(Q\\P x ) + H(Q))} VxeT Q , 

Px{T Q ) > (n + l)-Wexp{-nD(Q\\P x )}, 
Px(T Q ) < exp{-nD(Q\\P x )}. 

3 Previous results 

This section formulates the coding problem investigated in this paper, and shows 
a fundamental bound of the coding rate. We again note that previous work [TT] 
considered lossy coding, whereas this paper considers lossless coding. 

First, we formulate the coding problem of the complementary delivery coding 
system. 

Definition 1. (Fixed-to-fixed complementary delivery (FF-CD) code) [TT] 
A set (<p™, ifim, °f an enc °der and two decoders is an FF-CD code with 

parameters (n, M n , e n , e„ ) for the source (X,Y) if and only if 

^ : i Mn x y n - X n , : i Mn xx n ^y n , 

e (f ) = Pr ^ X n } , e { P = Pr {y u ^ F 11 } , 

where 

- def 

Y n d = £ ^ 2) (p n (X n ,Y n ),X n ). 
Definition 2. (FF-CD-achievable rate) 

R is an FF-CD-achievable rate of the source (X, Y) if and only if there exists a se- 
quence {(<£>", ffi\y <P?2))}ri=i of FF-CD codes with parameters {(n, M n , e*^, en )} 
for the source (X, Y) such that 

lim sup — log M n < R 7 

n — >oo ri 

lim = lim e ( P = 0. 

n — >oo n — >oo 

Definition 3. (Inf FF-CD-achievable rate) 

Rf(X,Y) = inf{R : 

R is an FF-CD-achievable rate of (X, Y)}. 

Willcms, Wolf and Wyner [T2J [TT] investigated a coding problem where sev- 
eral users are physically separated but communicate with each other via a satel- 
lite, and determined the minimum coding rate for the three users when trans- 
mitting to and from the satellite. The complementary delivery coding system 
is a special case of the system described by Willems et al., which considers the 
case of two users. Therefore, we can immediately obtain the closed form of 
Rf(X, Y) from the result obtained by Willems et al. 
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Theorem 1. (Coding theorem of FF-CD codes) 

R f (X,Y) = max{H(X\Y),H(Y\X)} 

= mnx{H(P xlY \P Y ),H(P Ylx \P x )} 

4 Code construction 

This section shows an explicit construction of universal codes for the comple- 
mentary delivery coding system defined by Definition [T] The coding scheme is 
described as follows: 
[Encoding] 

1. Determine a set S n (R) of joint types as 

S n (R) = {Qxy E V n {X x y) : 

max{H(V\Q x ),H(W\Q Y )} < R, 

Qxy = QxV = Q Y W, 

Ve V n (y\Q x ),We V n (X\Q Y )}, 

where R > is a given coding rate. We note that the joint type Q XY 
specifies the types Q x , Q Y , and the conditional types V and W. 

2. Create a table (henceforth we call this a coding table, sec Figure [2] left) for 
each joint type Q XY £ S n (R). Each row of the coding table corresponds to 
a sequence x £ Tq x , and each column corresponds to a sequence y £ Tq y . 

3. Mark cells that correspond to sequence pairs (x,y) £ Tq xy ( see Figure [2| 
middle). Codewords will be given only to sequence pairs that correspona 
to marked cells. 

4. Fill the marked cells with exp(ni?) different symbols such that each symbol 
occurs at most once in each row and at most once in each column. An 
example of symbol filling is shown on the right in Figure [2] right. 

5. For a given pair of sequences (x, y) 6 X x y with the joint type Q XY , if 
Q XY £ S n (R), the index assigned to the joint type Q XY of (x,y) is the 
first part of the codeword, and the symbol filling the cell of (x, y) in the 
coding table of Q XY is determined as the second part of the codeword. For 
the sequence pairs (x, y) whose joint type Q XY does not belong to S n (R), 
the corresponding codeword is determined arbitrarily and an encoding 
error is declared. 

[Decoding: <P(\)] (Almost the same as for ^ 2 )) 

1 . Find the coding table of the type Q XY that corresponds to the first part 
of the received codeword. The decoder can find the coding table used in 
the encoding scheme if no encoding error occurs. In this case, Q XY should 
be Q XY . 

2. Find the cell filled with the second part of the received codeword from the 
column of the side information sequence y £ Tq y ■ The sequence x £ Tq x 
that corresponds to the row of the cell found in this step is reproduced. 
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Figure 2: Example of coding scheme (left) Coding table (middle) Positions 
where codewords will be provided (right) Provided codewords 



First, we show the existence of such coding tables. To this end, we introduce 
the following two lemmas. 

Lemma 4. For a given coding table of a joint type Qxy € V n (X x y), the 
number of marked cells in every row of the coding table, N v (Qxy), is a constant 
value that is less than exp(ni?) ; and the number of marked cells in every column 
of the coding table, N x (Qxy), is also a constant value of less than exp(nR), 
both of which depend solely on the joint type Qxy- 

Proof. Note that the number of marked cells in each row equals the cardinality 
of the V-shell Ty(x) for the sequence x S Tq x that corresponds to the row. 
The cardinality of V-shells Ty(x) is constant for a given joint type Qxy and 
any sequences x € Tq x , this cardinality is bounded as follows: 

N y (QxY) = \T?(x)\ 

< cxp{nH(V\Q x )} (•.• Lemma [2j 

< exp(ni?). 

In the same way, the number of marked cells in each column equals the cardi- 
nality of the V-shell T\y(y) for the sequence y £ Tg y that corresponds to the 
column, and therefore it can be bounded as 

N v (Qxy) = \Tw(y)\ < exp(nR). 
This concludes the proof of Lemma [4j □ 

Lemma 5. For given integers m x , m y , n x and n y that satisfy m x > n x and 
m y > n y , there exists an m x x m y table filled with max(n x ,n y ) different symbols 
such that 

• at most n y cells are filled with a certain symbol for each row (blank cells 
are possible), 

• at most n x cells are filled with a certain symbol for each column (blank 
cells are possible), 

• each symbol occurs at most once in each row and at most once in each 
column. 



G 



•^Tj "^4 




Proof. The table mentioned in this lemma is equivalent to a bipartite graph 
such that 

• each node in one set corresponds to a row in the table, and each node in 
the other set corresponds to a column in the table, 

• each edge corresponds to a cell in the table, to which a certain symbol is 
assigned, 

• max(n a; , n y ) different colors are given to edges, each of which corresponds 
to a symbol in the table, 

• no two edges with the same color share a common node. 

Figure [3] shows an example of such a graph. Here, let us introduce the following 
lemma for bipartite graphs: 

Lemma 6. (Konig [T51 [TFj ) 

If a graph G is bipartite, the minimum number of colors necessary for edge 
coloring of the graph G equals the maximum degree of G. 

Lemma [6] ensures the existence of the above bipartite graph. This concludes the 
proof of Lemma [5] □ 

From Lemmas [4] and [5] we can easily show the existence of coding tables by 
setting m x = \TQ X \, m v = \T$ Y \, n x = \Tfr{y)\ and n y = \T${x)\ in Lemma|5| 

5 Coding theorems 

We can obtain the following theorem for the universal FF-CD codes constructed 
in Section [3] The proof is similar to that of the theorem of universal coding for 
a single source. 
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Theorem 2. For a given real number R > 0, there exists a sequence of universal 

(X) (Y) 

FF-CD codes with parameters {(n, M n , e„ , e„ such that for any integer 

n > 1 and any source (X,Y) with a generic distribution Pxy S M(X x 3^) 



-logM n < P+-|# xy|log(n+l), 
n n 

e {x) + e (y) 




n min £>(Q XY || Pxy) ^ 

Qxr65„(fl) J 

w/iere S„(P) =? B (Afxy)- S n (P). 

Proof. Lemmas [4] and [5] ensure the existence of a coding table for every joint 
type Qxy £ P(A' x y). From the coding scheme, the size of the codeword set 
is bounded as 

M n < \V n (X x y)\cxp(nR) 

< (n + l) lXxyl exp(nP), Lemma|TJ 

which implies the first inequality of Theorem [2] Next, we evaluate decoding error 
probabilities. Since every sequence pair (x,y) 6 Tq xy that satisfies Qxy & 
S n (R) is reproduced correctly at the decoder, the sum of error probabilities is 
bounded as 

+ e (y) 

< 2Pr {3Qxy eS n (R), (X n ,Y n )eT% XY } 

< 2 eM-nD(Q XY \\PxY)} 

Qx Y eS n (R) 
(•.■ Lemma [3]) 

-n min D(Q XY \\Pxy) \ 

Qxy£S"(R) 




n min D(Q X y\\Pxy) 

Qxy £S" (R) 



This completes the proof of Theorem [2] □ 

We can see that for any real value R> Rf (X, Y) we have 
min D{Qxy\\Pxy) > 0. 

QxY£S n (R) 

This implies that any real value R > Rf(X, Y) is a universal FF-CD achievable 
rate of (X, Y) , namely, there exists a sequence of universal FF-CD codes with 
parameters {(n, M„, e„ , e„ )}J£Lj such that 

lim sup — log M n < P, 

n — >oo n, 
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lim el x) = lim e { P = 0. 

n — >oo n — >oo 

The following converse theorem indicates that the error exponent obtained 
in Theorem [2] is tight. 

Theorem 3. Any sequence of FF-CD codes with parameters {(n, M n , e\\\ en)}^—\ 
for the source (X, Y) must satisfy 

xexp<-n min D{Qxy\\Pxy) f 

{ QxyeS n (R+e n ) J 

for any integer n > 1 and a given coding rate R = l/nlogM n > 0, where 

e n d ^ -{|A-xy|log(n+l) + l} 
n 

— > (n — > 00). 

Proof. Note that the number of sequences to be decoded correctly for each 
decoder are at most exp(ni?). Here, let us consider a joint type Qxy € S n (R + 
£„). Lemma [2] and the definition of S n (R+ e n ) imply that for (x,y) G Iq xy 
we have 

max{\TZ(x)\,\T&(y)\} 

> (n + lp x ^ 

Xmax[exp{nH(V\Q x )},exp{nH(W\Q Y )}} 

> (n + lp Xxy \cxp(n(R+e n )) 
= 2exp(ni?). 

Therefore, at least half of the sequences in the V-shell Ty(x) will not be decoded 
correctly at the decoder <p™ 2 )> or a ^ l eas t half of sequences in the V-shell T^>(y) 
will not be decoded correctly at the decoder f^iy Thus, the sum of error 
probabilities is bounded as 

e W + e< y ) 

> \ E Pr{(X n ,Y n )eT QxY } 

Q X Y&S n (R+e n ) 

> I( n+ 1)-I*x*| 

x E exp{-nC(g X y||P X y)} 

Qxy£S„(JJ+f„) 

(•.■ Lemma IS]) 



x exp < —n min D(Q X y \\Pxy) 

Qx Y eS n (R+e n ) 



This concludes the proof of Theorem [3j □ 
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The following corollary is directly derived from Theorems [2] and [3j 
Corollary 1. For a given real number R > 0, there exists a sequence of uni- 

( X) (Y) 

versal FF-CD codes with parameters {{n,M ni e n , e„ )}^=i such that for any 
source (X, Y) 

lim sup — log M n < R, 

n — >oo ri 

lim --log( e W + e^)= min D{Q XY \\P XY ), 

Qxy£S(R) 



n— >oo n 



where 

S{R) = {Qxy e M(X x y) : 

m ax {H(V\Q x ),H(W\Q Y )} < R, 

Qxy = QxV = Q y W, 

V € M(y\Q x ),W € M(X\Q y )}, 

and S(R) = M(X x y) - S(R). 

In a similar manner, we can investigate the probability such that the original 
sequence pair is correctly reproduced. The following theorem shows the lower 
bound of the probability that can be attained by the proposed coding scheme. 

Theorem 4. For a given real number R > 0, there exists a universal lossless 
f-FCD code {(</?«, , <P^)}'^Li such that for any integer n > 1 and any DMS 
(X,Y) 

-logM„ < R+e n , 
n 

l-Ce^ + e^) > exp(-n(e n + min D(Q XY \\P XY ) 

Proof. The first inequality is derived in the same way as the proof of Theorem[2] 
Next, we evaluate the probability such that the original sequence pair is correctly 
reproduced. Since every sequence pair (x,y) € Tq xy that satisfies Qxy G 
T n (R) is reproduced correctly at the decoder, the sum of the probabilities is 
bounded as 

> Pr{pT,y") G T% XY : Q X y G T n (R)} 

> (n+lp x ^exp{~nD(QxY\\PxY)} (1) 

Qx Y eT"{R) 

> (n + lH^^expi-ra min D(Q XY \\Pxy) 

I Qx Y eT"{R) 



= exp <-n[e n + min D(Q X y\\Pxy) 
where Eq. (|T|) comes from Lemma [3] This completes the proof of Theorem 

m □ 
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The following converse theorem indicates that the error exponent obtained 
in Theorem [4] might not be tight. 

Theorem 5. Any lossless f-FCD code {(<p n , ffl , (p [ n ] )}n=i f° r the DMS (X, Y) 
must satisfy 

< exp< — n[ —e n + min 

I- V Q XY ev n (Xxy) 

\m&x(H(V\Q x ),H(W\Q Y )) — (R+ e n )\ + + D(Q XY \\P XY )) } 

for any integer n > 1 and a given coding rate R — l/n log M n > 0. 

Proof. Note that the number of sequences to be decoded correctly for each 
decoder are at most exp(ni?). Here, let us consider a joint type Qxy & V n {X X 
y) such that Q XY = Q X V = Q Y W, V e M{y\Q x ) and W e M(X\Q Y )}. The 
ratio r c (Qxy) of sequences in the sequence set Tq xy such that the sequences 
are correctly reproduced is at most 

rdQxv) 

. [ , ( exp(nR) exp(nR)\ 

< mm < mm -— — ; — -r- , — — ; — -r , 1 

< min[cxp(ni?) • (n + l)\ Xxy \ exp{-nma.x{H{V\Q x ), H(W\Q Y ))}, ll(2) 

(3) 

= min exp{n(R + j n )}exp{-nma,x(H(V\Q x ),H(W\Q Y ))},l 

= exp {-n \max{H(V\Q x ), H{W\Q Y )} - (iZ + 7 „)| + } 

where Eq. ^ comes from Lemma [2j Therefore, the probability P c (Qxy) 
such that the original sequence pair with type Qxy is correctly reproduced is 
bounded as 

Pc(Qxy) 

< r c (Q XY )Pv{(X n ,Y n )€T^ xy } 

< exp{-n\max(H(V\Q x ),H(W\Q Y )) - (R + j n )\ + + D(Q XY \\P XY )}. 
Thus, the sum of the probabilities is obtained as 

l-(eW + eW) 

< E p c(Qxy) 

Qx Y ev n (xxy) 

< cxp[-n\m & x{H{V\Qx),H{W\Q Y )}-{R + ln)\ + + D{Qxy\\Pxy)] 

Q XY ev n (xxy) 

< exp\-n min (\max{H(V \Q X ), H(W\Q Y )\ 

Qxvev n (Xxy) y ' 

- (R + ln )\ + + D(Q XY \\P XY ))]. 
This completes the proof of Theorem [5] □ 
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We can see that for any real value R> Rf (X, Y) we have 

min \\max{H{V\Qx),H{W\Q Y )}-R\ + D{Qxy \\Pxy)) 

QxY&M{Xxy) u 1 

= \max(H(P Ylx \P x ),H(P xlY \P Y )) - R\ + 
= 0. 

On the other hand, for any real value R < Rf(X, Y) we have 
min D(Q XY \\P XY ) 

Qxy€T(R) 

> min (\max(H(V\Qx),H(W\Q Y ))-R\ + D(Q XY \\P XY )) \ 
QxYeM(xxyy A 

> 0. 

This implies that the error exponent obtained in Theorem [4] might not be tight. 



6 Variable-length coding 

This section discusses variable-length coding for the complementary delivery 
coding system, and shows an explicit construction of universal variable-length 
codes. The coding scheme is similar to that of fixed- length codes, and also 
utilizes the coding tables defined in Section [4] 

6.1 Formulation 

Definition 4. (Fixed-to-variablc complementary delivery (FV-CD) code) 

A set (ip n , <?™iy<P(2)) °f an enc °der and two decoders is an FV-CD code for the 

source (X, Y) if and only if 



¥>(2) 
AY) 



ip n (x n ,y n ) x y n -> 
ip n (x n ,y n ) x x n -» 

= PrjjC"^ X™} =0, 
= Pr<{Y" ^ F n } = 0, 



y n , 



where 



X n 



dcf. 
def. 



^(^(r,r),r), 

^ 2) ( V n (X n ,Y n ),X n ), 
and the image of ip n is a prefix set. 
Definition 5. (FV-CD-achievable rate) 

R is an FV-CD-achievable rate of the source (X, Y) if and only if there exists a 
sequence of FV-CD codes {(ip n , <?(2))}nLi f° r the source (X, Y) such that 

\imsvv~E[l{ip n {X n ,Y n ))] < R, 

n — >oo Tl 

where l(-) : B* — > 1Z is a length function. 
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Definition 6. (Inf FV-CD-achievable rate) 

Rv(X,Y) =inf{R : 

R is an FV-CD-achievable rate of (X, Y)}. 

6.2 Code construction 

We can construct universal FV-CD codes in a similar manner to universal FF- 
CD codes. Note that the coding rate depends on the type of sequence pair to 
be encoded, whereas the coding rate is fixed beforehand for fixed-length coding. 
The coding scheme is described as follows: 
[Encoding] 

1. Create a coding table for each joint type Qxy G V n (X x y) in the same 
way as Step 2 of Section [4] 

2. Mark cells that correspond to sequence pairs (x,y) 6 Tq xy . 

3. Fill the marked cells on the coding table with different max{ \Ty (x) | , \Tfy (y) | } 
symbols such that each symbol occurs at most once in each row and at 
most once in each column, where x £ Tq x , y £ Tq y . 

4. For a given pair of sequences (x,y) £ X n x y n , the number (index) 
assigned to the joint type Qxy of (x, y) is the first part of the codeword, 
and the symbol filling the cell of (x,y) in the coding table of Qxy is 
determined as the second part of the codeword. 

[Decoding] 

Decoding can be accomplished in almost the same way as the fixed-length cod- 
ing. Note that the decoder can always find the coding table used in the encoding 
scheme. 

6.3 Coding theorems 

We begin by showing a theorem for (non-universal) variable-length coding, 
which indicates that the inf coding rate of variable-length coding is the same as 
that of fixed-length coding. 

Theorem 6. (Coding theorem of FV-CD code) 

Rv(X,Y) = Rf(X,Y) =max{H(X\Y),H(Y\X)}. 

Proof. See Appendix. □ 

The following direct theorem for universal coding indicates that the coding 
scheme presented in the previous subsection can achieve the inf achievable rate 
clarified in Theorem [6] 

Theorem 7. There exists a sequence of universal FV-CD codes {{<p n , <?(i)> ?(2))}ra= 
such that for any integer n > 1 and any source (X, Y), the overflow probability, 
namely the probability that the length of a codeword exceeds a given real number 
R > 0, is bounded as 

- Pn {R) ^ Pr{l(cp n (X n ,Y n ))>n(R+e n )} 
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< (n + l)l* x *l 

x exp \ -n min D(Q X y \\Pxy) \ , 
{ Qx Y eS n (R) J 

where e„ is defined in Theorem^ This implies that there exists a sequence of 
universal FV-CD codes {(<p n , $>m> ¥>(2))}n=i ^at satisfies 

limsup-Z^pC",^)) < R V (X,Y) a.s. (4) 

n — >oo Tl 

Proof. The overflow probability can be obtained in the same way as an upper- 
bound of the error probability of FF-CD codes, which has been shown in the 
proof of Theorem [2] Thus, from Theorem [6] we have 

^Pr[-/(^"(X",y rl )) > R v (X,Y) + s\ < 00 

n=l ^ 71 ' 

for a given 8 > 0. From Borel-Cantelli's lemma [T71 Lemma 4.6.3], we immedi- 
ately obtain Eq. Q . This completes the proof of Theorem [7] □ 

The following converse theorem for variable-length codingindicates that the ex- 
ponent of the overflow probability obtained in Theorem [7] is tight. This can be 
easily obtained in almost the same way as Theorem [3j 

Theorem 8. Any sequence of FV-CD codes {(<p n , finy <?(2))}n=i f or ^ e source 
(X, Y) must satisfy 

p n (R)>(n + ir lXxyl 

x exp \ -n min D(Q X y\\P XY ) \ ■ 

{ Qx Y eS n (R+e n ) J 

for a given real number R > and any integer n > 1, where e n is defined in 
Theorem^ 

The following corollary is directly derived from Theorems [7] and [8j 

Corollary 2. There exists a sequence of universal FV-CD codes ^(i)> ¥^(2))}n=i 
such that for any source (X, Y) 

limsup-% n pT\y")) < Ry{X,Y) a.s. 

n — >-oc Tl 

lim --\ogp n (R) = mm D(Qxy\\Pxy) 

™^°° n QxYeS(R) 

Next, we investigate the underflow probability, namely the probability that 
the length of a codeword falls below a given real number R > 0. For this 
purpose, we present the following two theorems. The proofs are almost the 
same as those of Theorems |4] and |5] 
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Theorem 9. There exists a universal lossless v-FCD code {(<^ nj ^Pn\ 'fn^y^Li 
such that for any integer n > 1 and any DMS (X, Y), the underflow probability 
P n {R) is bounded as 

p(R) d ^ Pr{Z(^„(X",y")) <nR} 



< exp -n e„| min D{Q X y \\Pxy, 

This implies that there exists a universal lossless v-FCD code {(</?„, <fn \ <fin ^)}5^=i 
that satisfies 

liminf > R V (X,Y) a.s. (5) 

n^oo 77, 

Theorem 10. Any lossless v-FCD code {((^ n ,^l 1) , (p\?)}^ =l for the DMS 
(X, Y) must satisfy 

P n (R) 

< exp< — n\ — e„(l) + min 

I- V QxYeVn(Xxy) 

\max(H(V\Q x ),H(W\Q Y )) - (R + e n )\ + + D(Q XY \\P XY )) } 
for a given real number R > and any integer n > 1 . 



7 Concluding remarks 

We invesigated a universal coding problem for the complementary delivery cod- 
ing system. First, we presented an explicit construction of universal fixed-length 
codes, which was based on a graph-theoretical technique. We clarified that the 
error exponent achieved by the proposed coding scheme is asymptotically opti- 
mal. Next, we applied the coding scheme to construction of universal variale- 
length codes. We clarified that there exists a universal code such that the 
codeword length converges to the minimum achievable rate almost surely, and 
that the exponent of the overflow probability achieved by the proposed cod- 
ing scheme is optimal. This paper dealt with only the lossless configuration, 
and therefore constructing universal lossy codes for the complementary delivery 
coding system still remains as an open problem. 
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A Proof of Theorem [6] 



A.l Direct part 

Proof. We can apply a sequence of achievable FF-CD codes (fixed-length codes) . 
The encoder tp n assigns the same codeword as that of the fixed-length code to a 
sequence pair (x,y) 6 X n x y n that is correctly reproduced by the fixed-length 
code. Otherwise, the encoder sends the sequence pair itself as a codeword. 

The above FV-CD code can always reproduce the original sequence pair at 
the decoders, and it attains the desired coding rate. □ 



A. 2 Converse part 

Proof. We can prove the converse part in a similar manner to that for fixed- 
length coding. Let a sequence {(<p n , iffiy <p? 2 ))}rv=i °f FV-CD codes be given 
that satisfies the conditions of Definitions [3Jand [5j From Definition [5] for any 
5 > there exists an integer n-y — n\{$) and then for all n > ni(S), we can 
obtain 

-E[l(ip n (X n ,Y n ))] < R + S. (6) 
n 

Here, let us define A n = <p n (X n ,Y n ). Since the decoder ffiy. can always re- 
produce the original sequence X n from the received codeword A n and side 
information Y n , we can see that 

H(X n \A n Y n ) = 0. (7) 

Similarly, we can obtain 

H(Y n \A n X n ) = 0. 
Substituting A n into Eq.([6]), we have 

n{R + 5) > E[l(A n )} 

> H(A n ) (•.■ A n is a prefix set) 

> H(A n \Y n ) 

> I(X n ;A n \Y n ) 

= H(X n \Y n ). (vEq.0) 

Since we can select an arbitrarily small S > for a sufficient large n, we can 
obtain 

R > -H(X n \Y n ) = H(X\Y). 
n 

In the same way, we also obtain 

R > H{Y\X). 

□ 
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